Evaluation Challenges for Bridging Semantic Gap: Shape Disagreements in the LIDC

Evaluating the success of prediction and retrieval systems depends upon a reliable reference standard, a ground truth. The ideal gold standard is expected to result from the marking, labeling, and rating of domain experts of the image of interest. However experts often disagree and this lack of agreement challenges the development and evaluation of image-based feature prediction of expert-defined "truth." This paper addresses the success and limitations in bridging the semantic gap between CT-based pulmonary nodule image features and the ratings of diagnostic characteristics recorded by expert pulmonary radiologists. The prediction of diagnostic characteristics promises to automatically annotate medical images with medically meaningful descriptions directly usable for indexing and retrieving in content-based image retrieval (CBIR) and assisting in computer aided diagnosis (CADx). Successful results in predicting texture characteristics will be contrasted with less successful results for boundary shapes. The two primary differences in agreement between radiologists will be discussed; the first concerns agreement about the existence of a nodule, while the second considers the variability in diagnostic ratings among radiologists who agree on the presence of a nodule.